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DETAILED ACTION 

1 . This Office Action is in response to an RCE filed by the Applicant on March 24, 
2008. 

2. The Office Actions of July 31 , 2007, February 6, 2007 and May 1 5, 2006 are 
incorporated into this Non-Final Office Action by reference. 

Status of Claims 

3. Claims 1,8,13 and 20 have been amended. Claims 1 6 and 1 7 have been 
previously cancelled. Claims 1-15 and 18-21 are pending on this application. 

Information Disclosure Statement 

4. The information disclosure statement has not been filed for this application. To 
comply with 37 CFR 1 .98(a)(1 ), the following is required: (1 ) a list of all patents, 
publications, applications, or other information submitted for consideration by the Office; 
(2) U.S. patents and U.S. patent application publications listed in a section separately 
from citations of other documents; (3) the application number of the application in which 
the information disclosure statement is being submitted on each page of the list; (4) a 
column that provides a blank space next to each document to be considered, for the 
examiner's initials; and (5) a heading that clearly indicates that the list is an information 
disclosure statement. 
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Claim Rejections - 35 USC § 101 

Claims 1-21 are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. The computer system must set forth a practical 
application of judicial exception to produce a real-world result. Benson, 409 U.S. at 71- 
72, 175 USPQ at 676-77. The invention is ineligible because it has not been limited to a 
substantial practical application. 

For a claimed invention to be statutory the claimed invention must produce a 
useful, concrete, and tangible result. As the Supreme Court has made clear, " [a]n idea 
of itself is not patentable," Rubber-Tip Pencil Co. v. Howard, 20 U.S. (1 Wall.) 498, 507 
(1874); taking several abstract ideas and manipulating them together adds nothing to 
the basic equation. In re Warmerdam, 31 USPQ2d 1754 (Fed. Cir. 1994). 

For a claimed invention to be statutory under 35 U.S.C. 101, the claims must 
provide a tangible result, and there must be a practical application, by either: 1) 
transforming (physical thing) or 2) by having the FINAL RESULT (not the steps) achieve 
or produce a useful (specific, substantial, AND credible), concrete (substantially 
repeatable/non-unpredictable), AND tangible (real world/non-abstract) result. 

A claim that recites a computer that solely calculates a mathematical formula is 
not statutory. 

In the present case, claim 1 describes a neural network system. The claim does 
not provide a useful result because the claimed subject matter fails to sufficiently reflect 
at least one practical utility set forth in the descriptive portion of the specification. A 
neural network, in and of itself, is useless in a real world situation absent a particular 
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substantial application. The claims are not limited to a substantial practical application 
because they encompass a neural network without any description as to what it does or 
what is the output obtained from the network. Simply stated, there is no specific 
purpose or use for the neural network system claimed. Claims that recite a computer 
that solely calculates a mathematical formula are not statutory 

The claim also fails to provide a tangible result because the claimed subject 
matter fails to produce a result that is limited to having real world value rather than a 
result that may be interpreted to be abstract in nature as, for example, a thought, a 
computation, or manipulated data. More specifically, the claimed subject matter 
provides for a neural network system. The claim does not provide a result from the 
system that has a real world value. 

Claims 2-7 depend from claim 1 and incorporate the same deficiencies, and 
furthermore fail to rectify the aforementioned deficiencies. 

Claims 8-1 2, 1 3-1 5 and 1 8, 1 9 and 20-21 recite limitations similar to that of 
claims 1-8 and are rejected on the same basis. 



Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a 
person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived 
by the manner in which the invention was made. 
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6. Claims 1 -1 5 and 1 8-21 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Kevin Gurney in view of Hatayama ("An Introduction to Neural 
Networks", referred to as Gurney; US Patent #6,219,657; referred to as Hatayama). 

Claim 1 

Gurney anticipates an interconnecting neural network system (Gurney: page 1 , 
L16-24) comprising: a neural network unit that includes a plurality of neurons (Gurney: 
page 1 , L1 6-24; page 2, L1 9 to page 3, L9; Figs 1 .2 or 1 .3), each of the neurons 
outputting an excitation strength according to a similarity between input vectors and 
centroid vectors based on a kernel function (Gurney: page 1, L16-24; page 2, L19 to 
page 3, L9; page 1 4, L1 -27; page 1 82, L1 5 to page 1 85, L21 ; Figs 1 .2, 1 .3, 2.4; 
Examiner's Note (EN): the activation (excitation strength) is calculated by the difference 
||x-w|| (similarity). Radial Basis Function is a kernel function. Moreover, any function 
used by the neuron to provide its activation is a kernel function); and network control 
unit that constructs an artificial neural network structure by interconnecting neurons 
relating to each other among the neurons in the neural network unit via a weight 
(Gurney: page 1, L16-24; page 2, L19 to page 3, L9; Figs 1 .2 or 1 .3; EN: learning will 
develop a final structure for the neural network. The weights will relate the nodes in the 
final structure. Note that no restriction has been placed on the claim as to what "relating 
to each other" entails), wherein each of the neurons in the neural network unit outputs 
an excitation strength according to the similarity between the input vectors and the 
centroid vectors based on the kernel function when each neuron is excited by the input 
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vector applied from an outside (Gurney: page 1, L16-24; page 2, L19 to page 3, L9; 
page 14, L1-27; page 182, L15 to page 185, L21; Figs 1.2, 1.3, 2.4; EN: depending on 
the input vector, activation (excitation) will be calculated. The input vector must be 
provided from somewhere), and outputs a pseudo excitation strength obtained based on 
an excitation strength output from the other neuron when each neuron is excited in a 
chain reaction to excitation of the other neuron connected to each neuron (Gurney: 
page 1 , L1 6-24; page 2, L1 9 to page 3, L9; page 1 4, L1 -27; page 1 82, L1 5 to page 1 85, 
L21; Figs 1 .2, 1 .3, 2.4; EN: the activation (excitation strength) of each neuron will be the 
input to a neuron in a successive layer (chain reaction). The activation calculated by 
the neurons or the RBF units are considered pseudo excitation strength as understood 
from paragraph 60 of the specification of the instant application), wherein each of the 
neurons in the neural network unit has a plurality of modalities different from one 
another (Gurney: pages 44-46, sections 4.4 to 4.5.2; page 182, L15 to page 185, L21; 
EN: the association units will provide each neuron the functionality of operating on 
different input data by dividing the input pattern into a grid and providing an output 
based on the input pattern, therefore providing different "modalities" for the neurons. 
Moreover, if the neuron can classify two classes, then it has a plurality of modalities, 
since it can provide a response for different input data. Moreover, each RBF will have a 
different centre (different basis function or "modality") and provide a contribution to the 
output). 

Gurney does not teach the plurality of modalities of the neurons including 
auditory modality and visual modality so that a plurality of different input vectors of 
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auditory modality and visual modality can be handled (as opposed to actually being 
handled) simultaneously and independently by the neurons. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that a plurality of different input vectors of auditory 
modality and visual modality can be handled simultaneously and independently by the 
neurons (Hatayama: C3, L31 to C4, L53; C6, L 43-47; Figs. 1, 2, or 13; EN: the neural 
network receives camera information (visual modality) and voice information (auditory 
information) as well as other data from the user and apparatus information (different 
modalities). The pieces of the information input to the neural network are vectors of 
different "modalities" or type of data. Therefore, a plurality of different input vectors are 
received by the neural network and handled simultaneously. A person of ordinary skill 
in the arts would also recognize that neural networks operate on data inputted in vector 
form and that each neuron will process the inputs received independently. Also note 
that the recitation that a function can be performed (as opposed to actually performing; 
this is not a positive limitation but only requires the ability to so perform. It does not 
constitute a limitation in any patentable sense). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that a 
plurality of different input vectors of auditory modality and visual modality can be 
handled simultaneously and independently by the neurons as taught by Hatayama for 
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the purpose of having a multifunction system capable of operating on different types of 
data (Hatayama: C1, L14-17). 
Claim 2 

Gurney anticipates each neuron in the neural network unit outputs the pseudo 
excitation strength and also outputs the centroid vector of each neuron when each 
neuron is excited in a chain reaction to the excitation of the other neuron connected to 
each neuron (Gurney: page 182, L15 to page 185, L21; EN: calculating the activation is 
calculating the activation is calculating the excitation strength. The centroid vector is the 
weight vector and it will be provided (outputted) to the next node connected (weighted 
connections) to the node). 
Claims 3 and 9 

Gurney anticipates the network control unit interconnects the neurons relating to 
each other among the neurons in the neural network unit, based on an order of the 
neurons added or excited at time series in association with a plurality of input vectors 
applied to the neural network unit from the outside (Gurney: page 1 , L16-24; page 2, 
L19 to page 3, L9; page 14, L1-27; Figs 1.2 and 1.3; EN: learning will develop a final 
structure for the neural network. During training, the neurons will be added (connected) 
or excited on a layer-by-layer basis. Since each neuron will have to wait to receive data 
from a neuron in a preceding layer, the interconnections will be made in time series). 
Claims 4 and 15 

Gurney anticipates the network control unit trains the weight that connects the 
neurons to each other, based on the excitation strength of each neuron in the neural 
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network unit (Gurney: page 1, L16-24; page 2, L19 to page 3, L9; page 4, L10-31; page 
14, L1-27; pages 39-44, sections 4.1-4.4; EN: learning is training. Each neuron will 
provide an output as input to the nodes to which it is connected). 
Claims 5 and 10 

The network control unit removes each neuron at a predetermined timing 
determined based on the excitation strength of each neuron in the neural network unit 
(Gurney: page 1 , L1 6-24; page 2, L1 9 to page 3, L9; page 4, L1 0-31 ; page 1 4, L1 -27; 
pages 39-44, sections 4.1-4.4; EN: the weights will determine the strength of the 
connections, therefore connecting or disconnecting (removing) neurons during training). 
Claim 6 

Gurney anticipates each neuron in the neural network unit is an intermediate 
layer neuron using, as the centroid vector, centroid data in a matrix form in light of time 
series changes, and each intermediate layer neuron is connected to an output layer 
neuron that outputs a change in the excitation strength output from each intermediate 
layer neuron at time series (Gurney: page 182, L15 to page 185, L21; Fig1 0.1 5; EN: a 
vector is considered a one row matrix. Training will update the weight vector (time 
series changes)). 
Claims 7, 11, 18 and 21 

Gurney teaches the kernel function employed in each neuron in the neural 
network unit includes a radial basis function (Gurney: page 182, L15 to page 185). 
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Claim 8 

Gurney anticipates a method of constructing an interconnecting neural network 
structure (Gurney: page 1, L16-24), the method comprising the steps of: preparing an 
artificial neural network structure including a plurality of neurons (Gurney: page 1, L16- 
24; page 2, L19 to page 3, L9; Figs 1.2 and 1.3), each of the neurons outputting an 
excitation strength according to a similarity between an input vector and a centroid 
vector based on a kernel function (Gurney: page 1, L16-24; page 2, L19 to page 3, L9; 
page 14, L1-27; page 182, L15 to page 185, L21; Figs 1.2, 1.3, 2.4; Examiner's Note 
(EN): the activation (excitation strength) is calculated by the difference ||x-w|| (similarity). 
Radial Basis Function is a kernel function. Moreover, any function used by the neuron 
to provide its activation is a kernel function), the neurons relating to each other 
interconnected in the artificial neural network structure via a weight (Gurney: page 1 , 
L1 6-24; page 2, L1 9 to page 3, L9; Figs 1 .2 and 1 .3; EN: learning will develop a final 
structure for the neural network. The weights will relate the nodes in the final structure); 
and training the weight that connects the neurons to each other, based on the excitation 
strength of each neuron (Gurney: page 1, L16-24; page 2, L19 to page 3, L9; page 4, 
L10-31; page 14, L1-27; pages 39-44, sections 4.1-4.4; learning is training. Each 
neuron will provide an output as input to the nodes to which it is connected), wherein 
each of the neurons in the artificial neural network structure has a plurality of modalities 
different from one another Gurney: pages 44-46, sections 4.4 to 4.5.2; page 182, L15 to 
page 185, L21; EN: the association units will provide each neuron the functionality of 
operating on different input data by dividing the input pattern into a grid and providing an 
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output based on the input pattern, therefore providing different "modalities" for the 
neurons. Moreover, if the neuron can classify two classes, then it has a plurality of 
modalities, since it can provide a response for different input data. Moreover, each RBF 
will have a different centre (different basis function or "modality") and provide a 
contribution to the output). 

Gurney does not teach the plurality of modalities of the neurons including 
auditory modality and visual modality so that a plurality of different input vectors of 
auditory modality and visual modality can be handled simultaneously and independently 
by the neurons. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that a plurality of different input vectors of auditory 
modality and visual modality can be handled simultaneously and independently by the 
neurons (Hatayama: C3, L31 to C4, L53; C6, L 43-47; Figs. 1, 2, and 13; EN: the neural 
network receives camera information (visual modality) and voice information (auditory 
information) as well as other data from the user and apparatus information (different 
modalities). The pieces of the information input to the neural network are vectors of 
different "modalities" or type of data. Therefore, a plurality of different input vectors are 
received by the neural network and handled simultaneously. A person of ordinary skill 
in the arts would also recognize that neural networks operate on data inputted in vector 
form and that each neuron will process the inputs received independently. Also note 
that the recitation that a function can be (as opposed to actually doing) performed is not 
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a positive limitation but only requires the ability to so perform. It does not constitute a 
limitation in any patentable sense). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that a 
plurality of different input vectors of auditory modality and visual modality can be 
handled simultaneously and independently by the neurons as taught by Hatayama for 
the purpose of having a multifunction system capable of operating on different types of 
data (Hatayama: C1, L14-17). 
Claims 12 and 19 

Gurney anticipates a computer readable recording medium storing an 
interconnecting neural network structure construction program that allows a computer to 
execute the method according to claim 8 (Gurney: page 1, L16-24; EN: Neural 
networks are computer systems. Computers must have programs stored in some 
medium in order to perform its operations). 
Claim 13 

Gurney anticipates a method of constructing a self-organizing neural network 
structure including a plurality of neurons (Gurney: page 1 15 to page 118, section 8.2; 
page 185, L12-21), each of the neurons outputting an excitation strength according to a 
similarity between an input vector and a centroid vector based on a kernel function 
(Gurney: page 182, L15 to page 185, L21; Fig. 2.4; Examiner's Note (EN): the 
activation (excitation strength) is calculated by the difference ||x-w|| (similarity). Radial 
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Basis Function is a kernel function), the neurons relating to each other being 
autonomously connected in the self-organizing neural network structure based on the 
input vector (Gurney: page 115 to page 122, section 8.2.1), the method comprising: a 
first step of adding a neuron, which has an input vector as a centroid vector for a kernel 
function, into the self-organizing neural network structure as a new neuron based on an 
input vector that is input first from an outside (Gurney: page 1 15 to page 122, section 
8.2.1 ; page 1 82, L1 5 to page 1 85, L21 ; EN: during training, lateral connections will be 
made based on the input vector); and a second step of repeating following processings 
(a) to (c), each of the processings being based on an input vector that is an n th input 
vector from the outside, where n is an integer equal to or greater than 2: (a) the 
processing of calculating excitation strengths of all the neurons in the self-organizing 
neural network structure based on the n th input vector input from the outside; (b) the 
processing of adding a neuron, which has the n th input vector as a centroid vector for a 
kernel function, into the self-organizing neural network structure as a new neuron in 
case that it is determined by the processing (a) that there is no neuron excited such that 
the excitation strength thereof exceeds a predetermined threshold, among one or a 
plurality of neurons in the self-organizing neural network structure; and (c) the 
processing of performing both of or one of formation of a weight that connects the 
neurons, and training of the formed weight based on the excitation strengths of the 
neurons in the self-organizing neural network structure (Gurney: page 1 15 to page 120, 
L25; page 185, L12-21); wherein each of the neurons in the self-organizing neural 
network structure has a plurality of modalities different from one another (Gurney: 
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pages 44-46, sections 4.4 to 4.5.2; page 182, L15 to page 185, L21; EN: the association 
units will provide each neuron the functionality of operating on different input data by 
dividing the input pattern into a grid and providing an output based on the input pattern, 
therefore providing different "modalities" for the neurons. Moreover, if the neuron can 
classify two classes, then it has a plurality of modalities, since it can provide a response 
for different input data. Moreover, each RBF will have a different centre (different basis 
function or "modality") and provide a contribution to the output). 

Gurney does not teach the plurality of modalities of the neurons including 
auditory modality and visual modality so that a plurality of different input vectors of 
auditory modality and visual modality can be handled simultaneously and independently 
by the neurons. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that a plurality of different input vectors of auditory 
modality and visual modality can be handled simultaneously and independently by the 
neurons (Hatayama: C3, L31 to C4, L53; C6, L 43-47; Figs. 1, 2, and 13; EN: the neural 
network receives camera information (visual modality) and voice information (auditory 
information) as well as other data from the user and apparatus information (different 
modalities). The pieces of the information input to the neural network are vectors of 
different "modalities" or type of data. Therefore, a plurality of different input vectors are 
received by the neural network and handled simultaneously. A person of ordinary skill 
in the arts would also recognize that neural networks operate on data inputted in vector 
form and that each neuron will process the inputs received independently. Also note 
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that the recitation that a function can be (as opposed to actually doing) performed is not 
a positive limitation but only requires the ability to so perform. It does not constitute a 
limitation in any patentable sense). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that a 
plurality of different input vectors of auditory modality and visual modality can be 
handled simultaneously and independently by the neurons as taught by Hatayama for 
the purpose of having a multifunction system capable of operating on different types of 
data (Hatayama: C1, L14-17). 
Claim 14 

Gurney anticipates in the second step, a processing (d) of removing a neuron 
determined to be unnecessary based on the excitation strengths of the neurons in the 
self-organizing neural network structure is further performed (Gurney: page 1 16 to 
page118, section 8.1; page 127 to page 129, section 8.3.3; EN: inhibitory connections 
will remove neurons. Moreover, if a neuron is "off' it has been removed). 

Claim 15 

Gurney anticipates each of the neurons in the self-organizing neural network 
structure holds a class label relating to a final output, and, in the processing (c) in the 
second step, only in case that the class label held by each neuron in the self-organizing 
neural network structure is identical, both of or one of the formation of the weight that 
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connects the neurons, and the training of the formed weight is performed based on the 
excitation strengths of the neurons (Gurney: pages 135-136, section 8.3.6). 
Claim 20 

Gurney anticipates an interconnecting neural network system comprising: a 
plurality of intermediate layer neurons (Gurney: page 71, section 6.6, L1-4; page 72, 
Figs. 6.4 and 6.5; page 184, L13-16; Fig 10.15), each of the intermediate layer neurons 
outputting an excitation strength according to a similarity between an input vector and a 
centroid vector based on a kernel function (Gurney: page 1, L16-24; page 2, L19 to 
page 3, L9; page 1 4, L1 -27; page 1 82, L1 5 to page 1 85, L21 ; Figs 1 .2, 1 .3, 2.4; 
Examiner's Note (EN): the activation (excitation strength) is calculated by the difference 
||x-w|| (similarity). Radial Basis Function is a kernel function) and each of the 
intermediate layer neurons using centroid data in a matrix form in light of time series 
changes as the centroid vector (Gurney: page 182, L15 to page 185; Fig. 10.15; EN: a 
vector is considered a one row matrix. Training will update the weight vector (time 
series changes)); an output layer neuron connected to each of the intermediate layer 
neurons and outputting a change in the excitation strength output from each 
intermediate layer neuron at time series (Gurney: page 1, L16-24; page 2, L19 to page 
3, L9; page 14, L1-27; page 182, L15 to page 185, L21); wherein each of the neurons in 
the self-organizing neural network structure has a plurality of modalities different from 
one another (Gurney: pages 44-46, sections 4.4 to 4.5.2; page 182, L15 to page 185, 
L21; EN: the association units will provide each neuron the functionality of operating on 
different input data by dividing the input pattern into a grid and providing an output 
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based on the input pattern, therefore providing different "modalities" for the neurons. 
Moreover, if the neuron can classify two classes, then it has a plurality of modalities, 
since it can provide a response for different input data. Moreover, each RBF will have a 
different centre (different basis function or "modality") and provide a contribution to the 
output). 

Gurney does not teach the plurality of modalities of the neurons including 
auditory modality and visual modality so that a plurality of different input vectors of 
auditory modality and visual modality can be handled simultaneously and independently 
by the neurons. 

Hatayama teaches the plurality of modalities of the neurons including auditory 
modality and visual modality so that a plurality of different input vectors of auditory 
modality and visual modality can be handled simultaneously and independently by the 
neurons (Hatayama: C3, L31 to C4, L53; C6, L 43-47; Figs. 1, 2, and 13; EN: the neural 
network receives camera information (visual modality) and voice information (auditory 
information) as well as other data from the user and apparatus information (different 
modalities). The pieces of the information input to the neural network are vectors of 
different "modalities" or type of data. Therefore, a plurality of different input vectors are 
received by the neural network and handled simultaneously. A person of ordinary skill 
in the arts would also recognize that neural networks operate on data inputted in vector 
form and that each neuron will process the inputs received independently. Also note 
that the recitation that a function can be (as opposed to actually doing) performed is not 
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a positive limitation but only requires the ability to so perform. It does not constitute a 
limitation in any patentable sense). 

It would have been obvious to one of ordinary skill in the arts at the time of the 
applicant's invention to modify the teachings of Gurney by incorporating the plurality of 
modalities of the neurons including auditory modality and visual modality so that a 
plurality of different input vectors of auditory modality and visual modality can be 
handled simultaneously and independently by the neurons as taught by Hatayama for 
the purpose of having a multifunction system capable of operating on different types of 
data (Hatayama: C1, L14-17). 



Response to Applicant's arguments 

7. The Applicant's arguments have been fully considered but are not persuasive. 

8. In reference to Applicant's arguments: 

Applicant respectfully submits that, in the present invention, the neurons in the 
neural network unit have a plurality of modalities different from each other. The plurality 
of modalities of the neurons including auditory modality and visual modality so that a 
plurality of different input vectors of auditory modality and visual modality can be 
handled simultaneously and independently by the neurons. Therefore, the neurons can 
handle a plurality of different input vectors each having a particular, different length, 
such as auditory data and visual data. In other words, according to the present 
invention, a plurality of different data such as the five senses of a human being (e.g., 
auditory sense, visual sense of a human being, etc.) can be inputted together to this 
neural network at the same time and processed by interconnecting the neurons in this 
system. 



Examiner's response: 

Item 14 applies. The claims make no mention of the length of the input vectors. 
Also note that any neuron in a neural network can only process data one at a time. 
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That is, a computer system, such as a neural network, can only process one instruction 
at any given time. As stated in the arguments above, the only way that data from 
different sources can be processed at the same time is by inputting them together. 
Therefore, as taught by Hatayama, receiving pieces of information (vectors) from 
different sources by the neural network will process these vectors of different 
"modalities" simultaneously (Hatayama: C3, L31 to C4, L53; C6, L 43-47; Figs. 1,2, 
and 13). 

9. In reference to Applicant's arguments: 

In particular, although Hatayama discloses that the user's information is collected 
by sensors such as a camera 1 or a microphone 2, such information is not directly input 
to the behavior determination block 53. More specifically, Hatayama in FIG. 5 discloses 
that the user's information is first processed by the input detection information 
processing block 51 (see col. 5, lines 15-22), and the processed information is 
subsequently sent to the behavior determination block 53 as a single vector, not a 
plurality of vectors as recited in claims 1, 8, 13 and 20. As shown in FIG. 8 of 
Hatayama, in the behavior determination block 53, the user's information is first 
processed to form a single vector (S201), and the processed information is 
subsequently sent to the neural network emotion state input step (S202)(see col. 5, 
lines 43-48). 

In addition, Hatayama nowhere discloses that the neurons in the neutral network 
unit have a plurality of modalities different from one another or that the plurality of 
modalities of the neurons including auditory modality and visual modality as recited in 
claims 1, 8, 13 and 20. 

Hatayama also fails to teach "the plurality of modalities of the neurons including 
auditory modality and visual modality so that a plurality of different input vectors of 
auditory modality and visual modality can be handled simultaneously and independently 
by the neurons" as recited in amended claims 1 , 8, 13 and 20. 

Examiner's response: 

Item 14 applies. As stated above, the neural network of Hatayama receives 
inputs from different sources, including a camera and a microphone. Unless there are 
sub-networks formed and each sub-network receives the different types or "modalities" 
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of data as inputs, the only way that all these different types of data can be processed at 
the same time by a neuron is by inputting them to the neural network as a single vector. 
Also note that nowhere in the claims is there any recitation of inputting data to the 
network. 

Examination Considerations 

10. Examiner has cited particular columns and line numbers (or paragraphs) in the 
references applied to the claims above for the convenience of the applicant. Although 
the specified citations are representative of the teachings of the art and are applied to 
specific imitations within the individual claim, other passages and figures may apply as 
well. It is respectfully requested from the Applicant in preparing responses, to fully 
consider the references in their entirety as potentially teaching all or part of the claimed 
invention, as well as the context of the passage as taught by the prior art or disclosed 
by the Examiner. The entire reference is considered to provide disclosure relating to the 
claimed invention. 

1 1 . The claims and only the claims form the metes and bounds of the invention. 
"Office personnel are to give the claims their broadest reasonable interpretation in light 
of the supporting disclosure. In re Morris, 127 F.3d 1048, 105455, 44USPQ2d 1023, 
1027-28 (Fed. Cir. 1997). Limitations appearing in the specification but not recited in the 
claim are not read into the claim. In re Prater, 415 F.2d, 1393, 1404-05, 162 USPQ 541, 
550-551 (CCPA 1969)" (MPEP p 2100-8, c 2, I 45-48; p 2100-9, c 1, I 1-4). The 
Examiner has full latitude to interpret each claim in the broadest reasonable sense. 
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Examiner will reference prior art using terminology familiar to one of ordinary skill in the 
art. Such an approach is broad in concept and can be either explicit or implicit in 
meaning. 

12. Examiner's Notes are provided with the cited references to prior art to assist the 
applicant to better understand the nature of the prior art, application of such prior art 
and, as appropriate, to further indicate other prior art that maybe applied in other office 
actions. Such comments are entirely consistent with the intent and spirit of compact 
prosecution. However, and unless otherwise stated, the Examiner's Notes are not prior 
art but a link to prior art that one of ordinary skill in the art would find inherently 
appropriate. 

13. Unless otherwise annotated, Examiner's statements are to be interpreted in 
reference to that of one of ordinary skill in the art. Statements made in reference to the 
condition of the disclosure constitute, on the face of it, the basis and such would be 
obvious to one of ordinary skill in the art, establishing thereby an inherent prima facie 
statement. 

14. Examiner's Opinion: items 1 1-13 apply. The claims and only the claims form the 
metes and bounds of the invention. The Examiner has full latitude to interpret each 
claim in the broadest reasonable sense. 

Conclusion 

1 5. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Huang US Patent #7,069,257 
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Pao et al. US Patent #6,907,412 
Hoya US PG PUB #2003/01 15165 
16. Claims 1-15 and 18-21 are rejected. 
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1 7. Any inquires concerning this communication or earlier communications from the 
examiner should be directed to Omar F. Fernandez Rivas, who may be reached 
Monday through Friday, between 8:00 a.m. and 5:00 p.m. EST. or via telephone at 
(571 ) 272-2589 or email omar.fernandezrivas@uspto.gov. 

If you need to send an Official facsimile transmission, please send it to (571) 273-2589. 

If attempts to reach the examiner are unsuccessful the Examiner's Supervisor, 
David Vincent, may be reached at (571 ) 272-3080. 

Hand-delivered responses should be delivered to the Receptionist @ (Customer 

Service Window Randolph Building 401 Dulany Street Alexandria, VA 22313), located 

on the first floor of the south side of the Randolph Building. 
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